# Speech Translation

TransVIP is an innovative voice-to-voice translation system developed by Microsoft Research that retains the speaker's voice characteristics and timing (i.e., rhythm and pauses) during the translation process, making it particularly useful for video dubbing scenarios. TransVIP achieves end-to-end inference through joint probability while utilizing multiple datasets for cascade processing. The main advantages of this technology include high adaptability, voice feature retention, and timing preservation, which provide significant value in multilingual communication and content localization.

ultravox-v0_4_1-mistral-nemo is a multimodal speech large language model (LLM) based on pre-trained Mistral-Nemo-Instruct-2407 and whisper-large-v3-turbo. The model can handle both speech and text input simultaneously, such as a text system prompt and a speech user message. Ultravox converts input audio into embeddings using a special <|audio|> pseudo-token and generates output text. Future versions plan to expand the token vocabulary to support generating semantic and acoustic audio tokens, which can then be input into a vocoder to produce speech output. The model is developed by Fixie.ai and is licensed under MIT.

fixie-ai/ultravox-v0_4_1-llama-3_1-8b is a large language model based on pre-trained Llama3.1-8B-Instruct and whisper-large-v3-turbo, capable of processing speech and text input to generate text output. The model converts input audio to embeddings using a special <|audio|> pseudo-token and generates output text. Future versions plan to expand the token vocabulary to support semantic and acoustic audio token generation, which can then be used by a vocoder to produce speech output. The model performs excellently in translation evaluation and has no preference adjustment, making it suitable for scenarios such as voice agents, speech-to-speech translation, and speech analysis.

Whisper is a general-purpose speech recognition model. It is trained on a large and diverse set of audio data and is a multi-task model capable of performing multilingual speech recognition, speech translation, and language identification.

Azure AI Speech Studio is a speech service platform that provides speech-to-text and text-to-speech capabilities. It helps applications achieve the ability to listen, understand, and communicate via speech. Speech Studio offers a variety of speech functionalities, including speech-to-text, real-time speech transcription, batch speech transcription, custom speech recognition, speech translation, and text-to-speech. Users can choose the appropriate functionalities based on their needs and quickly get started using sample code. Speech Studio also provides learning resources, such as documentation, quick start guides, Microsoft Q&A, and Microsoft Learn.

Leveraging our advanced algorithms, translate your videos into multiple languages while keeping the original voiceover intact. Save time and resources, and make your videos reach a global audience.